Evolutionary Pressures on Enterprise Data Management

Andy Peyton, Data Management Association, will speak on "Evolutionary Pressures on Enterprise Data Management", at the Australian National University in Canberra, 11am, 2 September 2013.

Evolutionary Pressures on Enterprise Data Management

Andy Peyton (Data Management Association)

COMPUTER SCIENCE SEMINAR Relational Databases

DATE: 2013-09-02
TIME: 11:00:00 - 11:45:00
LOCATION: CHEM T1
CONTACT: mohammad.yamin@anu.edu.au

ABSTRACT:
The amount of data collected and maintained by enterprises is increasing at near exponential rates. Traditional approaches to the management of data are unlikely to be able to support the enterprise requirements. As a result there are significant shifts occurring in database technologies, data management, and data-oriented careers. However the end-state is not really understood as we are yet to master the data volumes expected to be managed. The security of data is becoming increasingly important to large enterprises and government. A recent US investigation estimates that over $300 billion worth of intellectual property is lost each year as a result of theft, and much of that intellectual property is simply enterprise data that has been copied. Cloud computing has its implications for managing the data storage explosion, but for government agencies there are important considerations of possible third party access to data that should remain under strict access control. The Australian Government places a very high priority on the security of citizensa data that it holds. This presentation will address all the above issues in the context of large Australian enterprises for whom data is a key asset of the organisation.
BIO:
Andy Peyton has over 35 years experience as an IT Professional. He started his career developing software for military aircraft, radar systems, and naval weapon systems. He then moved into the consulting field where he supported broad range of government departments designing computer systems, assessing data quality problems, running IT projects, and designing large databases. He is currently a senior solutions architect at IP Australia which is the Australian Government agency that administers intellectual property rights and legislation relating to patents, trade marks, designs, and plant breeder rights. He has a Bachelor of Science degree from the University of Sydney, and a Master of Management Economics from the University of NSW. He is currently President of the Canberra Chapter of the Data Management Association (DAMA) and was formerly Chairman of the Canberra Branch of the Australian Computer Society (ACS).

ANU Data Management Interest Group

Greetings from the inaugural meeting of the ANU Data Management Interest Group, at the Australian National University House in Canberra.

ANU Data Commons

ANU is building an ANU Data Commons for research data. Such a repository is useful for maintaining data beyond on research project, while allowing the researcher to retain control of their data. This work comes out of the Australian National Data Service (ANDS).

A data repository is normally thought of as something the researcher manually deposits data into manually. However, scientific instruments commonly now have network connections and can log data automatically, directly into the repository.

The project is based on Fedora Commons Repository Software, which is widely used by academia and public institutions (not to be confused with the Fedora implementation of Linux). ANU is adding a web interface for Fedora Commons. The system uses Dublin Core type metadata descriptions.

It occurs to me that research projects are typically not confined to one institution and cooperation is useful. My suggestion would be to rename the project the International Data Commons, providing services to the research community, in the same way the ANU hosts supercomputer facilities and archives for the research community.

ANU Archives

The ANU Archives holds University Archives and of business, trade unions, professional associations and industry bodies, in Australia and the region of interest to researchers. Notable collections are the Noel Butlin Archives Centre, Pacific Research Archives, Map Collection, Photographs and Publications. The archive is computerizing its metadata and providing this to the National Library of Australia Trove Service. One of the quirky but useful collections the ANU Archive has is the records of hotel bars in NSW.

Making Ocean Data More Usable

Various local, national and international research bodies have oceanographic data repositories. The International Council for Science (ICSU) formed the Scientific Committee on Oceanic Research (SCOR) to help link up this data. Dublin Core standards were expanded to accommodate oceanographic needs. Details are in "Workshop on Data Publication" (SCOR/IODE/MBLWHOI Library, Workshop Report No. 230), which includes a MBLWHOI repository workflow and a Micro Life Cycle of Data.

Australian National Data Service

The Australian National Data Service (ANDS), is an Australian Government funded project, to help make research data available for long term use. As well as the Research Data Australia Discovery Service to find data, ANDS works on policy development and education, in particular on Ethics, consent and data sharing.

Australian Governments Open Access and Licensing Framework

The Australian Governments Open Access and Licensing Framework (AusGoal), provides guidance on use f open access licenses by Australian Governments. AusGoal is managed by the Cross Jurisdictional Chief Information Officers' Committee (CJCIOC), supported by the federal Department of Finance. AusGoal's approach provides a model for researchers to license data.

ANU Data Management Courses

The ANU offers research students a Data Management Workshop (ILDM01). There is also a fifty page ANU Data Management Manual (Managing Digital Research Data at the Australian National University, 2012).

Australian Data Archive

The Australian Data Archive (ADA) grew out of the Social Sciences Data Archive, using the national data faculties hosted at the ANU. The New OAIS architecture, is shown in "The Australian Data Archive: preserving the past for access in the future" by Dr. Steve McEachern
(Slide 9, COMPASS seminar series, March 2012). The ADA home page gets a W3C mobileOK Checker 80% score (which is very good) but the metadata does not appear to be exported to Trove.

ANU Research Data Management Policy

ANU's research data management policy is set out in sections 2.1 to 2.4 of the "Responsible Practice of Research Policy" (ANU, 22 November 2010).

Digital Humanities Hub Data Management

The ANU Digital Humanities Hub has an Online Cultural Collections Analysis and Management System (OCCAMS), as described in "The Data Management Journey". The The AUSTLANG system combines data on Aboriginal and Torres Strait Islander languages.

Data Citation

Digital Object Identifiers (DOIs) are used for identifying scholarly papers and in the citation of the papers in others. DOIs and similar identifiers can be used for identifying research datasets and used to cite data. Datasets need not be open to be cited, the metadata acts as a proxy for data which can not be publicly navigable due to privacy, commercial or national security reasons. ANDS provides an easy visual overview with "Building a Culture of Data Citation".

Adopting the Internet Process for Data Repositories

At present the process of integration of data is ad-hoc and lacking in a strategic approach. It occurs to me that researchers could look to the Internet for some techniques for accelerating standardization and use. While the technical details underlying the Internet are well know, what is not appreciated is the international political process used to create and promote the standards. This process cut trough much of the red tape which held up the development of data network standards and as great a contribution to the Internet success as the technology itself.

The creators of the Internet and the web founded a number of organizations to foster its development, promotion and standards development. On 28 August 2012, several of the ordinations announced an agreement with the IEEE on standards development, in "Leading Global Standards Organizations Endorse ‘OpenStand’ Principles that Drive Innovation and Borderless Commerce" (IEEE, IAB, IETF, Internet Society and W3C, 28 August 2012).

I suggest that those interested in data repositories for research should adopt the political processes used successfully for the Internet, to develop standards and promote access to data. The pre-Internet processes currently being used will not be sufficient. It should be noted that Australia was closely involved in the development and implementation of the Internet and much of that actively took place in Australian universities, particularly the ANU. That expertise is available for advising on data repository promotion.

One way to promote data reuse is to consider the motivations of those involved. Researchers are motivated by funding and publication considerations. If data reuse is a criterion used by finding bodies to decide if research gets funded and if data repositories can be counted as publications and use of the data counted as a publication citation, then this will be a powerful motivation to do the extra work to make data available.

Academics tend to take a roundabout way to approach a topic. I suggest that if data repositories are to take hold a much more brief, direct and results orientated approach needs to be taken. Those promoting data repositories need to succinctly explain how to make a dataset cite-able and how to cite it, give successful example and explain the benefits in terms of better research metrics and grants. It is not necessary, nor desirable, to preference this with a long scholarly history of data management.

Data Management Graduate Courses at ANU

The first ANU Data Management Interest Group meeting, Australian National University, Canberra, 9am 4 September 2012. I will be speaking on "Data Management Graduate Courses at ANU".

Google Page Rank Sparks New Computing Paradigm

In Map-Reduce and its Children today at NICTA in Canberra, Professor Jeffrey Ullman discussed the wider uses for the approach Google uses for calculating "PageRank" for web sites:

Abstract: Since its publication by Google researchers in 2004, Map-reduce has proven to be a significant advance in programming methodology that offers resilient, easy-to-code parallel computation on a modern computing cluster or "cloud." It has led to a variety of systems that improve it in different ways. One direction is raising the level of abstraction, especially to support relational-database operations. A second direction is increasing the generality, while maintaining the programmability and resiliency in the face of partial failures. We shall review the environment of NICTA a map-reduce system, give some examples of how it works, and discuss the various extensions and the technical problems they posed.
From: Map-Reduce and its Children, NICTA, 2010

Professor Ullman started by introducing Distributed File Systems, as popularised by Google. He described how they are large (many Terabytes), divided into chunks (typically 64mbytes). These databases are usually added to (with few random updates). Chunks are replicated at compute nodes in racks connected by switches. Apart from the Google File System (GFS), HDFS and CloudStore.

On the distributed file system is built Map-Reduce, Key-Value stores and SQL. Map-reduce (such as Hadoop) is alongside the object store (key value pair).

Map reduce allows for large numbers of highly parallel processing tasks. It must allow for failure of nodes without having to restart the entire job. Previously failure could be though of as a rare event, but with tens of thousands of nodes, it is common.

Key-Value pair databases (a throwback to pre SQL relational databases?). SQL-like implementers are PIG, Hive, Sawzall and Scope.

Map-reduce has two functions: map and reduce. Large numbers of tasks are created for each to run in parallel and the results combined. Each map task operates on a distinct portion of the input file. The map tasks produce pairs of values. These are then passed to the reduce tasks, with pairs with the same "key" pass to the same reduce task, where they are combined. The result of the reduce tasks then are placed in the file store.

Clearly this design is intended for tasks such as indexing web pages for searching. It is good for matrix multiplication and relational algebra (such as a natural join). It assumes there are not many updated of data in place, just data added (that assumption works well for many web applications).

The process for dealing with a failed task is simple: restart the task. Professor Ulman commented that on real systems about 5% of failures are due to software problems (such as the version of Java not being upgraded).

Map reduce can be generalised to allow any number of tasks to be strung together. Despite the complexity, such a system can be designed to be fault tolerant: either the processing is still underway or it has finished successfully. This sounds good, but still seems to be at the experimental stage. The simpler map-reduce is easier to get to work as there are just two tasks before the results are safely stored in the file system.

It would be interesting to look at how generally applicable these applications are. Normally you would think these would work for web applications, such as indexing web pages, but not traditional databases, such as bank records. Financial records in a bank require frequent update in place of data (such as the current balance). However, banks have to keep transaction records. Normally these transaction records are seen as less important that the current updates and just something needed occasionally. But if the client has access to their bank account on-line, they are more likely to be looking at the history. This makes the transaction records important. Also "cloud" based applications may result in more aggregations of data which suit this approach.

The Professor argued that recursion is now key to web based applications, such as Page-Rank and web structure. However, this appears an area for research, not practical implementation.

Electronic Document and Records Management Standard

International Council on Archives issued "Principles and Functional Requirements for Records in Electronic Office Environments" (2008). This is now ISO 16175-3:2010 (25 November 2010). It would therefore seem something to teach in "Electronic Document and Records Management" course at ANU.

ICA are working on "A training package consisting of a customisable learning resource kit, pedagogical manuals, training exercises, powerpoint slides and supporting reading." but this not scheduled until 2012. So I will need to prepare my own materials (which might then be of assistance to National Archives of Australia, who are leading the ICA initiative).

"Principles and functional requirements for records in electronic office environments":

1. Module 1: Overview and Statement of Principles: background information, organisation, fundamental principles and additional context
2. Module 2: Guidelines and Functional Requirements for Electronic Records Management Systems: a global high-level statement of core and optional requirements, including application guidelines and a compliance checklist
   
3. Module 3: Guidelines and Functional Requirements for Records in Business Systems: guidelines and generic core and optional functional requirements for records in business systems.
   

Civil servants can use new ways to communicate

A seminar on Green IT in Sabah, Malaysia was held 30 December 2010. This was hosted by Sabah INTAN, Sabah Malaysia Administrative Modernisation and Management Planning Unit, Universiti Malaysia Sabah, state Computer Services Department and the Malaysian Cocoa Board. At this Datuk Muhammad Hatta Aziz, State federal secretary is reported to have said that most civil servants were still using paper mail and refusing to accept replies by email (at best allowing for fax). This wasted paper, as well as making government processes less efficient. It is despite the Malaysian government issuing a green technology policy last year.

The National Archives of Malaysia issued a comprehensive "Electronic Records Management System (ERMS) System Specifications for Public Offices" in 2009. This was developed into "Principles and Functional Requirements
for Records in Electronic Office Environments, Module 2" (International Council on Archives, 2008) now ISO 16175-3:2010 (25 November 2010) and so is comparable to those of other countries. This should allow Malaysian civil servants to communicate electronically.

The Faculty of Information Management at UiTM runs courses in e-records management. My own "Electronic Document and Records Management" course at ANU addresses this and is avialable online to students in Malaysia.

"IT-Sideways: Tech Blog Malaysia" reports that the only green measure in the 2011 budget was hybrid cars will enjoy .

Electronic Records Management at National Archives

National Archives of Australia (NAA) held a Government Agencies Information Network (GAIN) Forum in Canberra this morning. Barbara Berce from NAA began by introducing the GAIN initiative of NAA to communicate to agency staff. As well as face to face presentations, an online forum will be available in the next few weeks.

Barbara then reprised her presentation "From Red Tape and Reticence to Realisation" (from RMAA inForum 2010). She argued there are efficiency benefits from digital work practices and to "... stop living in the paper world".

NAA recently conducted a survey of government agencies. One question was if agencies had a comprehensive digital records management environment, with 30% saying yes, while 40% of the remainder having no plans to do so. Almost half of agencies have primarily paper records. Reasons for this included the difficulty of getting staff to use electronic records. A summary of the survey will be on the new GAIN web site.

Agencies are estimated to have 1,300 shelf kilometres of paper records. Much of this is outdated short term records and copies of records already stored electronically and so could be eliminated. Similar results have been found in the USA (testimony to congress by US government archivist) and New Zealand (archivists annual report).

One driver for change will be new FOI legislation. Agencies which cannot respond to FOIs quickly will have increased costs. Barbara pointed out the community now expects online and interactive access to government. This new information now needs to be retained.

The aim is that capture and disposal of records is automated. This was in "Note for file: A report on recordkeeping in the Australian Public Service" (Management Advisory Committee 8, from 2007). But apart from appropriate software (which is improving) this also requires organisational commitment. One point I disagree with the presentation on is cost. I suggest that better integrated records management could greatly reduce IT costs in agencies, rather than increasing it. As an example, eliminating the use of general purpose "office" software, particularly Microsoft Word, for the creation of documents would greatly reduce costs and also result in better records retention. Using web based software could eliminate most Ms-Word documents. Apart from the saving in licence costs (only about 10% of agency staff would need an actual copy of Ms-Office), this would would reduce storage requirements by 90% with much more efficient document formats used and and processor requirements by about 75%.

Barbara emphasised the cultural change needed. She suggested making the most of what equipment the agency has and considering records management requirements when new systems are acquired. But I suggest a culture change is needed at NAA, from thinking of themselves as being the ultimate repository of old records after they are no longer of use by agencies, to an active participant.

An example of how NAA could take a more active role would be to sponsor an online "cloud" system for smaller agencies. In conjunction with AGIMO a commercial provider would offer a complete system. All data in the system would be held in Australian Government approved facilities, under the control of Australian nationals on Australian territory. A small agency would then just need a credit card to pay for an account on the system, a web browser for each employee using it and a secure link to the cloud system. The agency would not need to purchase any records management or other software, nor run any servers of its own. Such a system could come complete with free open source software for running the basic operations of a government agency, including records management, word processing spreadsheets and email. NAA would then have a showcase for good electronic record practice, as well as an easy solution to the problem of supporting the archive needs of many small, and sometimes short lived, agencies with orphan record formats.

RMAA Canberra Meeting

The NAA meeting was followed by a meeting of the Records Management Association of Australasia (RMAA) Canberra branch Lunch Bag Education Update Seminar. I was invited to stay for the meeting and talk a little about the course COMP7420 "Electronic Data Management" I am designing for ANU.

This was an impromptu talk so I put the course web page up on screen and made these points:

1. I am not a records management professional, but an IT person who chaired a committee on electronic document management when a public servant. I designed a short intensive course run at ANU (one of my former students was at the RMAA session) and have been teaching records management and metadata to IT and business students.
2. ANU is the new kid on the block in terms of RM courses. It is not offering a full program at this point, just a couple of subjects (the other is COMP7410: Data Mining and Matching by Dr Peter Christen). These subjects are six weeks (half a standard university unit). The student needs to spend the equivalent of a least one hour a day studying.
3. Courses can be done one off, or contribute to a certificate, diploma or masers, with courses from across the campus as part of the ANU Graduate Studies Select program.
4. These courses are based in the Engineering and Computer Science with close ties to the cutting edge of new digital systems. As an example, David Hawking who developed the Funnelkack web search engine used by the Australian Government is an adjunct in the next office. We don't need electronic communications as we can shout through the wall. ;-)
5. Courses are using advanced e-learning techniques and the most advanced Australian developed software (Moodle).
6. Assignments will be work based: "wrote a report for your executive on the current practices in your organisation ... make recommendations for improving practices in your organisation.

Apart from being able to promote my coruse to potential students, the RMAA event was very useful as it presented details of the University of Canberra, CIT and other records management and information management courses. One point this made was that RM courses are spread across different disciplines at different universities and even in the one university.
One interesting point was that the NAA also runs courses, which while aimed at the pubic service, are available to the private sector. NAA also has teaching materials and e-learning modules.There was a discussion of professionalism which was similar to the discussions I have had at the ACS. A problem is that records managers are seen as people who push bits of paper around in the basement, rather than their role as information managers.
ps: NAA e-meeting room: Archives have equipped their meeting room with a hi-tech interactive lectern. The computer functions provided are very good, but like many such installations, the ergonomics could be better. The pen operated display screen has been mounted in the centre of a standard sloped lectern surface. There is a cordless mouse and compact keyboard provided. It would be better if the lectern was replaced with a more horozontal surface, with the screen towards the back. This would provide more space for the keyboard and more room for the mouse. The screen could be placed on a shelf with room for the keybaord to be pushed out of the way underneath. A larger, easier to operate keyboard could then be used. But that is a minor quibble and the equipment worked very well.

NSW unable to archive digital information

The NSW Auditor-General has reported to Parliament that the State Records Authority of New South Wales is in breach of its enabling legislation as it is "...unable to archive digital records despite growing reliance on the use of digital records by New South Wales Government agencies". The AG has recommended a government digital archiving solution be established. The AG also notes that the archive will run out of storage space during this financial year (e-records would reduce the space problem).

Help is at hand. The NSW State Records Authority provides a list of courses available on records management. First in the list is my own "Electronic Data Management" COMP7420, being offered at the Australian National University from January 2010. As I am still designing the course, I could include the requirements for the NSW e-Archive as a case study. Several students on my Green ICT course wrote official carbon reduction strategies for their agencies as their assignments. NSW government staff could write the e-archive strategy as their assignment for this course.

It should be noted that there is considerable guidance for implementing an e-arching system in NSW:

1. M2009-11 Premier's Memoranda: NSW Standard on Digital Recordkeeping
2. Standard on digital recordkeeping and Short guide to implementing the Standard
3. M2004-14 Use and Retention of Email for Government Communications
4. Policy on digital records preservation
5. Policy on electronic recordkeeping
6. ICT Attitudes to records and recordkeeping: survey report
7. Considering digital recordkeeping as part of systems development or procurement (RIB52)
8. Strategies for documenting Government business: The 'DIRKS' manual
9. Introducing recordkeeping metadata (RIB 18)
10. Selecting records management software (RIB 2)
11. Checklist for assessing business systems (RIB 42)
12. Managing digital records
13. Desktop Management: Managing electronic documents and directories (RIB 30)
14. Destroying digital records: When pressing delete is not enough (RIB 51)
15. Digital imaging and recordkeeping (RIB 11)
16. Email and other templates for creating digital records
17. FAQs - Emails and recordkeeping (RIB 49)
18. General Retention and Disposal Authority - Source Records That Have Been Migrated (GA 33)
19. Guidelines on keeping web records
20. Keeping recruitment records using e-Recruitment (RIB 56)
    
21. Managing the message: Guidelines on managing formal and informal communications as records
22. Records management and web 2.0
23. Archives Advice: Protecting and handling magnetic media (NAA)
24. Archives Advice: Protecting and handling optical disks (NAA)
25. Digital records preservation in the NSW public sector: A discussion paper
26. Australasian Digital Recordkeeping Initiative (ADRI) website
27. Digital Records Advisory Group

Ministerial Memorandum 2009-11 ‘Standard on Digital Recordkeeping’ acknowledged the increasing reliance upon digital records by New South Wales public sector agencies. The Memorandum also noted the benefits of a standardised approach to digital record keeping.

The Authority’s ‘Future Proof’ strategy aims to help government agencies manage digital records. A key aspect of the strategy is to implement a digital archiving facility for New South Wales government. Currently, the Authority declines digital records that an agency wishes to transfer for archiving because it does not have the infrastructure to access the information in the record.

The Crown Solicitor has advised that if the Authority holds public records that are not accessible, the Authority is in breach of its enabling legislation. Notwithstanding its current policy of declining records, the Authority holds some inaccessible records and is in breach.

Data standards for superannuation

The Super System Review Panel delivered its final report to the Government on 30 June 2010. It recommended "electronic transmission of linked financial and member data using standardised formats" (Super System Review Final Report, Chapter 9). This generated little interest in the media, but in a speech yesterday review chairman Jeremy Cooperis reported to have called for the government to enforce a standard as the superannuation industry has failed to implement one voluntarily.

The review recommendations, including data formats, are estimated to cost $1 billion to implement, but will then save $1 billion per year (The $20 billion prize: An industry blueprint to implement SuperStream, Joint Financial Services Council and Ernst & Young research, August 2010).

From Super System Review Final Report, Chapter 9:

5.2 Standardising data transmission

While some submissions argued that it is enough for the data to be provided, a clear majority favoured mandating a uniform standard format for delivery; that is, prescribing exactly the manner in which data is required to be provided. While this would promote confidence and clarity, the Panel is mindful of the cost involved in updating IT systems and existing member details to comply with any proposed changes.

Recommendation 9.4

APRA should convene a stakeholder group including at least the ATO, employers, payroll providers, super administrators and trustee representatives to devise online forms covering all the common processes between:

(a) the employer and the fund;
(b) the fund and the member; and
(c) different funds, such as occurs with ‘rollovers’.

The Panel considers that such forms should be adopted by all APRA-regulated funds, including for transactions involving rollovers to or from SMSFs, by January 2012.

There would also need to be standards applying to the wholesale (peer-to-peer) transactions that occur from fund to clearing house, clearing house to clearing house and clearing house to fund. ...

6.2 Achieving e-commerce as the norm in superannuation

The Panel is convinced that major cost savings are available in the superannuation industry through a shift from the fragmented and largely manual processing of member accounts, contributions and rollovers to a standardised electronic approach.

While attempts have been made to devise industry standards to facilitate this, they have been voluntary and to date have failed due to excessive complexity and low take up. The Panel notes the recent announcement by three major fund administrators of agreement on a set of principles to govern the electronic processing of rollovers between participating funds. These are to be further developed consistently with standards to be adopted by the Medicare Australia clearing house and using an open governance structure along the lines of that provided by the Australian Payments Clearing Association for banking.

Many administrators and clearing houses already engage with payroll providers to partially automate employer to fund transactions but, due to the lack of common standards across the industry, these
processes often require funds to provide specific software to large employers and/or the application of proprietary middleware solutions to convert the output from payroll providers into a format useable by the fund administrator.

The Panel believes that a key pre-condition to fully effective e-commerce in super is the availability of a data base containing accurate and secure details of all funds other than SMSFs. Details to be incorporated would include, as a minimum, fund name, SPIN and bank account details including name, BSB and account number. To avoid the extraction of monopoly profits, this data base needs to be governed and administered on a cost recovery basis, either by a collaborative industry enterprise or by a government agency. Given that most of the required data is already held by APRA, and that APRA already has mechanisms for secure electronic communication with funds and the SBR hub, the Panel considers that APRA is best placed to develop and administer the data base.

Recommendation 9.8

Treasury should convene a working group comprising representatives of relevant segments of the financial sector to devise the process for development of SBR-compatible standards that provide for linked personal and financial data transmission and facilitate related software development. The standards should address transactions between employer and fund, fund and member, and between funds.

Development work should be financed through an industry levy.

All administrators and clearing houses should be required to adopt these standards as a licence
condition. ...

From: Super System Review Final Report, Chapter 9, Super System Review Panel , 30 June 2010

Turning Laws into Decisions Online

The ACS Canberra branch meeting last night was "Taming Complex Legislation and Policy: A Business-Friendly IT Approach" about the product "Oracle Policy Manager". This is software originally developed in Canberra by Softlaw, later Ruleburst and acquired by Oracle in 2008. It is a very Canberra product, being originally designed to automate bureaucratic decision making in the Australian Government. It is now used internationally for government and commercial applications.

The core idea of policy maker is to help people make consistent decisions based on complex legal and policy rules. Laws and policy documents are converted into executable business rules. The business rules are still in a human readable format, essentially a form of structured natural language. These rules can then be run using a software interpreter to make a decision. The rules can be used by an employee who enterws the data in response to questions from a client in per4son, via a telephone or from a form. Alternqatively the data for decision making can be from a databse. The most interesting implementation is where the client themselves interact with the system via a web interfaqce.

As an example the Swedish government use the system to decide on payment of dental treatment refunds. The UK government system use the system to decide who is an employee and who is a contractor for tax purposes. A key aspect of this is that the tax payer can use the system themselves and see the result. An example which goes beyond simply paying money, is the NSW system for professionals, such as doctors, reporting children at risk of abuse.

The Australian Government Visa Wizard is implemented using the system. In a more complex case, it is used by the US IRS.

A key part of the system is that along with an answer, the system can explain how it came to that conclusion.

What is interesting is that the system is essentially designed to take legal and bureaucratic documents and semi-automatically convert them into something in a very well defined format which a computer can understand. Obviously it would be better if those writing laws did not
use natural language and instead authored directly in a rigorously structured language and then generated the natural language version. But this would require retraining those who write the laws. The system can be used to identify ambiguities in existing laws and policies.

One advantage not mentioned in the presentation of using an automated system to give citizens advice is confidentiality. It is well established that people will tell a computer program things which they are not comfortable to tell a person.

All the examples given were of public sector uses, but the product is also used by insurance companies, retail and universities.

US Government e-Record Management

The United States Government Accountability Office gave evidence to the US House of Representatives on "The Challenges of
Managing Electronic Records". The conclusion was that more needs to be done on e-record management in the US Government. GAO found almost 80 percent of agencies were at moderate to high risk of improper disposition of records. The situation in Australia is likely to better, as the Australian Government has more sophisticated policy and technical systems for handling e-records than the USA.

Why GAO Did This Study
Federal agencies are increasingly using electronic means to create, exchange, and store information, and in doing so, they frequently create federal records: that is, information, in whatever form, that documents government functions, activities, decisions, and other
important transactions. As the volume of electronic information grows, so does the challenge of managing electronic records. Both federal agency heads and the
National Archives and Records Administration (NARA) have responsibilities for managing federal records. ...

From: "The Challenges of Managing Electronic Records", Statement of Valerie C. Melvin, Director, Information Management and Human Capital Issues, Testimony Before the Subcommittee on Information Policy, Census and National Archives, Committee on Oversight and Government Reform, House of Representatives, Government Accountability Office, United States, June 17, 2010

Google Wave Real-time Document Collaboration

According to news reports, Google has acquired AppJet so that its EtherPad real-time document collaboration tool can be incorporated in Google Wave. I was not impressed with Google Wave's interface and offering EtherPad's fmailar wordprocessor like interface on top will be an improvement. The idea is to allow several eople to edit the same document at the same time and see the changes each other are making in real time. I will be incuding some of this in my new ANU course "Electronic Data Management" (COMP7420).